Image retrieval method and apparatus independent of illumination change

ABSTRACT

An image retrieval method and apparatus independent of an illumination change are provided. The image retrieval method involves: inputting a query image; detecting an illumination color from the query image and converting the illumination color into a standard illumination color; extracting color information of the query image by using color descriptors; and retrieving a similar image by comparing the extracted color information with color information of a database which converts a variety of images into images of standard illumination colors and extracts and stores the color information of the images in advance. According to the method, without changing the structures of color descriptors or a similarity comparison using color information, an image retrieval independent of illumination changes is enabled by adding only a preceding step of standard illumination conversion.

[0001] This application claims the priority of Korean Patent ApplicationNo. 2002-58462, filed Sep. 26, 2002, in the Korean Intellectual PropertyOffice. This application also claims the benefit of U.S. ProvisionalApplication No. 60/333,132, filed Nov. 27, 2001 and U.S. ProvisionalApplication No. 60/359,309, filed Feb. 26, 2002. The entire contents ofthese applications are incorporated herein by reference.

BACKGROUND OF THE INVENTION

[0002] 1. Field of the Invention

[0003] The present invention relates to an image retrieval method, andmore particularly, to an image retrieval method and apparatusindependent of an illumination change.

[0004] 2. Description of the Related Art

[0005] There are a variety of image retrieval methods. One of them is atext-based image retrieval method in which text is attached to an imageand by accessing the text, a desired image is retrieved. When an imageis retrieved using text, there are limitations in the text expressionsused to describe a picture. Also, semantic interpretations of an imagevary for different users who retrieve the image such that casesfrequently occur where an image totally different from an image which auser desires to retrieve is retrieved and output.

[0006] Another method is a content-based image retrieval method in whichthe major characteristic features of an image desired to be retrievedare used in retrieving the image. Major characteristic features in animage used in the content-based image retrieval method include a color,texture, shape, motion information, etc. When an image is retrievedusing these major characteristic features in the image, the probabilitythat an image desired to be retrieved by a user is found increases.Accordingly, this method can reduce retrieval errors that occur due tothe semantic disagreement in text-based image retrieval.

[0007] In color-based retrieval, which is a type of content-basedretrieval method, a similar image is found by using information on thedistribution, kinds, or locations of colors of pixels in an image amonga variety of information in the image. A leading retrieval method incolor-based retrieval is an image retrieval method using MPEG-7 colordescriptors which are the current standards adopted by the ISO/IEC15938-3. The color descriptors are broadly broken down into four types:a dominant color descriptor, a color structure descriptor, a colorlayout descriptor, and a scalable color descriptor.

[0008] The dominant color descriptor is used when a predetermined coloror a small number of colors represent the feature in the entire image orin a part of the image.

[0009] The color structure descriptor uses local color structureinformation together with a histogram. When characteristic informationis extracted, color information on each pixel is not extractedindependently, but in consideration of color information on neighboringpixels.

[0010] The color layout descriptor indicates the spatial distribution ofa color. Assuming that a unit obtained by dividing an x-y planeuniformly by a predetermined size is a bin, the spatial distribution ofcolors existing in an image can be expressed by the number of bins witha 64-bit value. Since similarity calculation can be performed in a verysimple way when the color layout descriptor is used, retrieval can beperformed quickly. Also, a natural image or a color sketch image can beused as a query. Accordingly, this color layout descriptor can beappropriately applied in video browsing and retrieval.

[0011] The scalable color descriptor shows a qualitative expression of acolor histogram encoded by a Harr transform, and uses a hue saturationvalue (HSV) color space. Since the similarity between two images isdetermined by the hamming distance between scalable color descriptors,similarity calculation can be performed quickly.

[0012] However, the image retrieval method using color informationexpressed by using the MPEG-7 color descriptors described above issensitive to illumination. Consequently, images having identicalcontents may have different color information due to small differencesof surrounding illuminations when the images are photographed such thatimage retrieval is not accurately performed. For example, a naturalimage may not be accurately retrieved due to a color information changecaused by time and weather changes (for example, brightness and shadowsin an image), a color information change caused by differentilluminations (for example, an incandescent lamp or a fluorescent lamp),or a color information change occurring when an identical image iscaptured by different image capture apparatuses (for example, camerasproduced by different manufacturers).

SUMMARY OF THE INVENTION

[0013] Accordingly, the invention provides an image retrieval method andapparatus independent of illumination changes, in which in order toretrieve an image independently of illumination changes, a query imageand images stored in an image database are converted into images ofillumination colors under a standard illumination and color informationof the converted images is compared to each other.

[0014] In one aspect, the invention provides an image retrieval methodindependent of illumination changes, comprising: inputting a queryimage; detecting an illumination color from the query image andconverting the illumination color into a standard illumination color;extracting color information of the query image by using colordescriptors; and retrieving a similar image by comparing the extractedcolor information with color information of a database which converts avariety of images into images of standard illumination colors andextracts and stores the color information of the images in advance.

[0015] In another aspect, the invention provides an image retrievalapparatus independent of illumination changes, comprising: a query imageinput unit which receives a query image; a standard illumination colorconverting unit which detects an illumination color from the query imageand converts the illumination color into a standard illumination color;a query image color information extracting unit which extracts colorinformation of the converted query image by using color descriptors; adatabase which converts a variety of images desired to be retrieved intoimages of standard illumination colors and extracts and stores the colorinformation of the images in advance; and a similar image retrieval unitwhich finds a similar image by comparing the color information of thequery image that is converted into an image of the standard illuminationcolor, with color information on the variety of images stored in thedatabase.

[0016] In another aspect, the invention provides a standard illuminationcolor converting method, comprising: removing low luminance parts andself luminance parts from an input query image; detecting anillumination color of the image from the remaining pixels after theremoval of the low luminance parts and self luminance parts; andconverting the detected illumination color into a standard illuminationcolor. Preferably, the detecting step, comprising: converting atri-stimulus value XYZ of each pixel of the query image from which lowluminance parts and self luminance parts have been removed, intochromaticity coordinates (x, y); projecting the converted chromaticitycoordinates on an x-y plane; dividing the x-y plane uniformly into agrid of a predetermined size; counting the number of projected pixelsexisting in each divided grid; removing pixels in a grid if the countingresult indicates that the counted value of the grid is less than apredetermined value; and calculating an average value of centralcoordinates of the remaining grids after the removal, and determiningthe average as a standard illumination color of the input image.

[0017] In another aspect, the invention provides computer readable mediahaving embodied thereon computer programs for performing theabove-described image retrieval method and the standard illuminationcolor converting method.

BRIEF DESCRIPTION OF THE DRAWINGS

[0018] The above objects and advantages of the present invention willbecome more apparent by describing in detail preferred embodimentsthereof with reference to the attached drawings in which:

[0019]FIG. 1 is a flowchart of an image retrieval process of the presentinvention;

[0020]FIG. 2 is a flowchart of a step for detecting an illuminationcolor in a query image and converting the color into a standardillumination color;

[0021]FIGS. 3A through 3D are diagrams for explaining a process forconverting a color into a standard illumination color;

[0022]FIG. 4 is a diagram of an embodiment of an expression form of acolor descriptor for describing an image;

[0023]FIGS. 5A and 5B show retrieval results before and after using animage retrieval method of the present invention;

[0024]FIG. 6 shows examples of outdoor natural images used in anexperiment;

[0025]FIG. 7 shows examples of set images of objects photographedindoors used in an experiment;

[0026]FIG. 8 is a block diagram of an image retrieval apparatusperforming the image retrieval method described above; and

[0027]FIG. 9 is a detailed block diagram of a standard illuminationcolor converting unit.

DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0028] Referring to a flowchart of an image retrieval process of thepresent invention in FIG. 1, first, a query image and a request toretrieve an image independently of illumination changes are input by auser in step S110. An illumination color is detected in the query imageand the color is converted into a standard illumination color in stepS120. Color information of the query image is extracted by using a colordescriptor in step S130. By comparing the extracted color informationwith color information in a database which converts a variety of imagesinto images of standard illumination colors and extracts and stores thecolor information of the images in advance, a similar image is searchedfor in step S140.

[0029] Referring to FIG. 2, the step S120 for detecting an illuminationcolor in a query image and converting the detected color into a standardillumination color will now be explained in detail. FIG. 2 is aflowchart of the step S120 for detecting an illumination color in aquery image and converting the color into a standard illumination color.

[0030] First, in the input query image, low luminance parts and selfluminance parts are removed in step S210. From the remaining pixels, theillumination color of the image is detected in step S220 and thedetected illumination color is converted into a standard illuminationcolor in step S230.

[0031] Step S210 for removing low luminance parts and self luminanceparts in the input query image is performed as follows. First, red,green, and blue (RGB) values of pixels of the input query image areconverted into a Commission Internationale de L'clairage (CIE)tri-stimulus value XYZ.

[0032] Then, by using the converted tri-stimulus value XYZ, pixelshaving low luminance values are removed. There are a number of methodsfor determining whether or not a pixel has a low luminance value. Forexample, if a Y value of a pixel is included in the lowest 5% of the Yvalues of all of the pixels, it may be determined that the pixel has alow luminance value.

[0033] Next, by obtaining a self luminous threshold, self luminancepixels in the image are removed. The self luminance pixel refers to apixel of an image of a luminescent object, such as the sun or electriclights. The self luminous threshold can also be obtained by using priorart methods, and pixels having values greater than the threshold aredetermined as self luminance pixels and removed.

[0034] The step S220 for detecting the illumination color of the inputimage from the remaining pixels is performed as follows. First, thetri-stimulus value XYZ of each pixel in the query image in which the lowluminance parts and self luminance parts are removed is converted intochromaticity coordinates (x, y). For example, the x value ofchromaticity coordinates may be calculated by a formula such asX/(X+Y+Z), while the y value of chromaticity coordinates may becalculated by a formula such as Y/(X+Y+Z).

[0035]FIGS. 3A through 3D are diagrams for explaining a process forconverting a color into a standard illumination color. Referring toFIGS. 3a through 3 d, the converting process will now be explained indetail.

[0036] First, the converted (x, y) values are projected on an x-y plane.Then, the figure as shown in FIG. 3A is obtained. The thus-obtained x-yplane is divided uniformly into bins of a predetermined size as shown inFIG. 3B. For example, the x-y plane may be divided into a 60×60 gridplane. Then, the number of pixels in each bin is counted and stored. Ifthe number is less than an arbitrary threshold, the bin is removed.

[0037] Then, the result in FIG. 3C is obtained. Next, by dividing thesum of central coordinate values of the remaining bins by the number ofremaining bins, an average chromaticity (x_(av), y_(av)) 310 is obtainedand is determined as the illumination color of the input image. Inparticular, x_(av) is obtained by dividing the sum of the x centralcoordinate values of the remaining bins by the number of remaining bins,and y_(av) is obtained by dividing the sum of the y central coordinatevalues of the remaining bits by the number of remaining bins.

[0038] Finally, the step S230 for converting the detected illuminationcolor into a standard illumination color will now be explained. First,standard illumination chromaticity coordinates (x_(c), y_(c)) 320 on thedaylight locus corresponding to a standard color temperature 6500K isfound. Then, by using the average chromaticity (x_(av), y_(av)) 310, thestandard illumination chromaticity coordinates (x_(c), y_(c)) 320, and aBradford color adaptation transform matrix, the tri-stimulus value XYZof each pixel of the input image is converted into a tri-stimulus valueX′Y′Z′ of the standard illumination. Finally, the thus-convertedstandard illumination tri-stimulus value X′Y′Z′ is converted into an RGBvalue.

[0039] In step S130 for extracting color information of the query imageby using a predetermined color descriptor, the variety of colordescriptors described above are used. That is, an image is described byusing the MPEG-7 standard color descriptors for expressing the colors ofan image in the query image and database images.

[0040]FIG. 4 is a diagram of an embodiment of an expression form of acolor descriptor for describing an image. This embodiment containsinformation on the four color descriptors.

[0041] In the final step S140 for searching for a similar image bycomparing the extracted color information with color information in adatabase which converts a variety of images into images of standardillumination colors and extracts and stores the color information of theimages in advance, the retrieval results are output in order ofincreasing distance, by calculating distances between extracted colordescriptors.

[0042]FIGS. 5A and 5B show retrieval results before and after using animage retrieval method of the present invention.

[0043] When a prior art image retrieval method was used and a commandfor retrieving images very similar to a postbox image was given, imagestotally different from the postbox image were output. However, when theimage retrieval method of the present invention was used, accurateresults were obtained as shown in FIG. 5b.

[0044] Tables 1a through 1d show gain changes when the image retrievalmethod of the present invention was used. TABLE 1a Not convertedConverted into into standard standard illumination illumination Gainchanges color color (ANMRR) Using parameters 0.183983 0.298456 +0.114473Not using parameters 0.325878 0.409459 +0.083581

[0045] TABLE 1b Not converted into Converted into Gain standardillumination standard illumination changes color color (ANMRR) Using0.429638 0.306655 −0.122983 parameters Not using 0.469480 0.340987−0.128493 parameters

[0046] TABLE 1c Not converted into Converted into Gain standardillumination standard illumination changes color color (ANMRR) Using0.710394 0.355556 −0.354838 parameters Not using 0.739785 0.389964−0.349821 parameters

[0047] TABLE 1d Not converted into Converted into Gain standardillumination standard illumination changes color color (ANMRR) Using0.488618 0.409816 −0.078802 parameters Not using 0.566681 0.464205−0.102476 parameters

[0048] Table 1a is the result when a CCD data set defined in the MPEG-7standard was used, and shows that when the retrieval method independentof illumination according to the present invention was used, gainincreased by 0.11 when parameters were used and by 0.08 when parameterswere not used. This result was expected because images having identicalcontents with different illuminations are rarely included in the CCDdata set itself.

[0049] Table 1b is the result when natural outdoor images as shown inFIG. 6 were used. Table 1c is the result when set images of objectsphotographed indoors as shown in FIG. 7 were used. Table 1d is theresult when the MPEG-7 CCD data set and outdoor natural images were usedtogether. Referring to tables 1b through 1d, it is shown that theretrieval method of the present invention is superior.

[0050]FIG. 8 is a block diagram of an image retrieval apparatusperforming the image retrieval method described above.

[0051] The image retrieval apparatus comprises a query image input unit810, a standard illumination color converting unit 820, a query imagecolor information extracting unit 830, a database 840, and a similarimage retrieval unit 850.

[0052] The query image input unit 810 receives a request to retrieve animage independent of illumination changes and a query image from a user.

[0053] The standard illumination color converting unit 820 detects anillumination color in the query image and converts the color into astandard illumination color. For this, low luminance parts and selfluminance parts are removed, the illumination color of the image isdetected from the remaining pixels, and the detected illumination coloris converted into a standard illumination color. These processes havebeen described in more detail above.

[0054] The query image color information extracting unit 830 extractscolor information of the query image by using predetermined colordescriptors. That is, an image is described by using the MPEG-7 standardcolor descriptors which express colors of the query image and databaseimages.

[0055] The database 840 converts a variety of images desired to beretrieved into standard illumination colors in advance and extracts andstores the color information of the images.

[0056] The similar image retrieval unit 850 finds a similar image bycomparing the color information of the query image that is convertedinto a standard illumination color with color information of a varietyof images stored in the database.

[0057]FIG. 9 is a detailed block diagram of the standard illuminationcolor converting unit 820.

[0058] A noise removing unit 910 removes low luminance parts and selfluminance parts in the input query image. An illumination colordetection unit 920 detects the illumination color of the image from theremaining pixels after the noise removing unit 910 removes noise. Aconverting unit 930 converts the detected illumination color into astandard illumination color.

[0059] The present invention may be embodied in a code, which can beread by a computer, on a computer readable recording medium. Thecomputer readable recording medium can be any kind of recordingapparatuses on which computer readable data are stored.

[0060] The computer readable recording media includes storage media suchas magnetic storage media (e.g., ROM's, floppy disks, hard disks, etc.),optically readable media (e.g., CD-ROMs, DVDs, etc.) and carrier waves(e.g., transmissions over the Internet). Also, the computer readablerecording media can be scattered on computer systems connected through anetwork and can store and execute a computer readable code in adistributed mode.

[0061] Although the present invention has been described with referenceto the embodiment above, the present invention is not limited to theembodiment described above, and it is apparent that variations andmodifications by those skilled in the art can be effected within thespirit and scope of the present invention defined by the appendedclaims. Therefore, the scope of the present invention is not determinedby the above description but by the accompanying claims.

[0062] As described above, without changing the structures of colordescriptors or the similarity comparison method in the prior artretrieval method using color information, the present invention enablesimage retrieval independent of illumination changes by adding only apreceding step of standard illumination conversion. That is, a method bywhich identical images photographed under different illuminations can beretrieved when an image is retrieved by using color information isprovided. According to this method, identical images which the userphotographed at different time points, under different weatherconditions, different illuminations or with different photographingapparatus can be effectively retrieved.

What is claimed is:
 1. An image retrieval method independent ofillumination changes comprising: inputting a query image; detecting anillumination color from the query image and converting the illuminationcolor into a standard illumination color; extracting color informationof the query image by using color descriptors; and retrieving a similarimage by comparing the extracted color information with colorinformation of a database which converts a variety of images into imagesof standard illumination colors and extracts and stores the colorinformation of the images in advance.
 2. The method of claim 1, whereinthe converting step comprises: removing low luminance parts and selfluminance parts from the input query image; detecting an illuminationcolor of the query image from the remaining pixels after the removal ofthe low luminance parts and self luminance parts; and converting thedetected illumination color into a standard illumination color.
 3. Themethod of claim 2, wherein the removing step comprises: converting ared, green, and blue (RGB) value of each pixel in the input query imageinto a tri-stimulus value XYZ; removing pixels having low luminancevalues from the pixels in the image, by using the converted tri-stimulusvalue XYZ; and removing self luminance pixels from the pixels in theimage, by using the converted tri-stimulus value XYZ.
 4. The method ofclaim 3, wherein if the Y value of the tri-stimulus value of a pixel isincluded in the lowest 5% of the Y values of all of the pixels, it isdetermined that the pixel has a low luminance value.
 5. The method ofclaim 3, wherein it is determined that pixels having values greater thana predetermined self luminance threshold are self luminance pixels, andthen the self luminance pixels are removed.
 6. The method of claim 2,wherein the detecting step comprises: converting a tri-stimulus valueXYZ of each pixel of the query image from which low luminance parts andself luminance parts have been removed, into chromaticity coordinates(x, y); projecting the converted chromaticity coordinates on an x-yplane; dividing the x-y plane uniformly into a grid of a predeterminedsize; counting the number of projected pixels existing in each dividedgrid; removing pixels in a grid if the counting result indicates thatthe counted value of the grid is less than a predetermined value; andcalculating an average value of central coordinate values of theremaining grids after the removal, and determining the average as astandard illumination color of the input image.
 7. The method of claim6, wherein the x value of the chromaticity coordinates is calculated bya formula X/(X+Y+Z) and the y value of the chromaticity coordinates iscalculated by a formula Y/(X+Y+Z).
 8. The method of claim 6, wherein thex-y plane is divided into a 60×60 grid.
 9. The method of claim 2,wherein the step for converting the detected illumination color into astandard illumination color comprises: finding standard illuminationchromaticity coordinate values on a daylight locus corresponding to astandard color temperature; converting the tri-stimulus value XYZ ofeach pixel of the illumination color detected in the detection step,into a standard illumination tri-stimulus value X′Y′Z′, by using thestandard illumination chromaticity coordinate values and a Bradfordcolor adaptation transform matrix; and converting the converted standardillumination tri-stimulus value X′Y′Z′ into an RGB value.
 10. The methodof claim 9, wherein the standard color temperature is 6500K.
 11. Themethod of claim 1, wherein in the extracting step, color information ofthe query image is extracted by using a dominant color descriptor when apredetermined color or a small number of colors in an image representthe entire image or a part of the image.
 12. The method of claim 1,wherein in the extracting step, color information of the query image isextracted by using a color structure descriptor which uses local colorstructure information together with a histogram, and when characteristicinformation is extracted, extracts color information on each pixel notindependently, but in consideration of color information on neighboringpixels.
 13. The method of claim 1, wherein in the extracting step, colorinformation of the query image is extracted by using a color layoutdescriptor which indicates the spatial distribution of a color.
 14. Themethod of claim 1, wherein in the extracting step, color information ofthe query image is extracted by using a scalable color descriptor whichshows a color histogram encoded by a Harr transform and uses a huesaturation value (HSV) color space.
 15. The method of claim 1, whereinin the retrieval step, retrieval results are output in order ofincreasing hamming distance by using the results of calculating hammingdistances between the extracted color descriptors.
 16. An imageretrieval apparatus independent of illumination changes comprising: aquery image input unit which receives a query image; a standardillumination color converting unit which detects an illumination colorfrom the query image and converts the illumination color into a standardillumination color; a query image color information extracting unitwhich extracts color information of the converted query image by usingcolor descriptors; a database which converts a variety of images desiredto be retrieved into images of standard illumination colors and extractsand stores the color information of the images in advance; and a similarimage retrieval unit which finds a similar image by comparing the colorinformation of the query image that is converted into an image of thestandard illumination color, with color information on the variety ofimages stored in the database.
 17. The apparatus of claim 16, whereinthe standard illumination color converting unit comprises: a noiseremoving unit which removes low luminance parts and self luminance partsfrom the input query image; an illumination color detection unit whichdetects an illumination color of the image from the remaining pixelsafter the noise removing unit has removed noise; and a converting unitwhich converts the detected illumination color into a standardillumination color.
 18. The apparatus of claim 17, wherein the noiseremoving unit converts a red, green, and blue (RGB) value of each pixelin the input query image into a tri-stimulus value XYZ, and removespixels having low luminance values and self luminance pixels from thepixels in the image, by using the converted tri-stimulus value XYZ. 19.The apparatus of claim 17, wherein the illumination color detection unitconverts a tri-stimulus value XYZ of each pixel of the query image fromwhich low luminance parts and self luminance parts have been removed,into chromaticity coordinates (x, y); projects the convertedchromaticity coordinates on an x-y plane; divides the x-y planeuniformly into a grid of a predetermined size; counts the number ofprojected pixels existing in each divided grid; removes pixels in a gridif the counting result indicates that the counted value of the grid isless than a predetermined value; and calculates an average of centralpoints of the remaining grids after the removal, and determining theaverage as a standard illumination color of the input image.
 20. Theapparatus of claim 17, wherein the converting unit finds standardillumination chromaticity coordinate values on a daylight locuscorresponding to a standard color temperature; converts the tri-stimulusvalue XYZ of each pixel of the illumination color detected in thedetection step into a standard illumination tri-stimulus value X′Y′Z′ byusing the standard illumination chromaticity coordinate values and aBradford color adaptation transform matrix; and then converts theconverted standard illumination tri-stimulus value X′Y′Z′ into an RGBvalue.
 21. A standard illumination color converting method comprising:removing low luminance parts and self luminance parts from an inputquery image; detecting an illumination color of the image from theremaining pixels after the removal of the low luminance parts and selfluminance parts; and converting the detected illumination color into astandard illumination color.
 22. The method of claim 21, wherein theremoving step comprises: converting a red, green, and blue (RGB) valueof each pixel in the input query image into a tri-stimulus value XYZ;removing pixels having low luminance values from the pixels in the imageby using the converted tri-stimulus value XYZ; and removing selfluminance pixels from the pixels in the image by using the convertedtri-stimulus value XYZ.
 23. The method of claim 21, wherein thedetecting step comprises: converting a tri-stimulus value XYZ of eachpixel of the query image from which low luminance parts and selfluminance parts have been removed, into chromaticity coordinates (x, y);projecting the converted chromaticity coordinates on an x-y plane;dividing the x-y plane uniformly into a grid of a predetermined size;counting the number of projected pixels existing in each divided grid;removing pixels in a grid if the counting result indicates that thecounted value of the grid is less than a predetermined value; andcalculating an average value of central coordinates of the remaininggrids after the removal, and determining the average as a standardillumination color of the input image.
 24. The method of claim 21,wherein the converting step comprises: finding standard illuminationchromaticity coordinate values on a daylight locus corresponding to astandard color temperature; converting the tri-stimulus value XYZ ofeach pixel of the illumination color detected in the detection step intoa standard illumination tri-stimulus value X′Y′Z′ by using the standardillumination chromaticity coordinate values and a Bradford coloradaptation transform matrix; and converting the converted standardillumination tri-stimulus value X′Y′Z′ into an RGB value.
 25. A computerreadable medium having embodied therein a computer program forperforming the method of claim
 1. 26. A computer readable medium havingembodied therein a computer program for performing the method of claim21.